Repository navigation
fix(engine): let users write what they ask about, and keep the app's context line out of their view - #165
Merged
Conversation
…anguage Every locale's simulated user is told to write in the locale's language, and English locales are told not to write phrases or sentences in another language; every locale's user-turn gate fails such a turn as a language violation. A user asking for a translation, or practising a language and asking for corrections, has to write the phrase in question, so general_open_ended's language_translation topic and general_educational's language-learning subject drew retries, n_user_language_violations and gate exhaustion. Every language rule now leaves that text alone. The user's instruction and its per-turn requirement say that text it asks about, such as a phrase to translate or a sentence to have corrected, may stay in its own language, and every gate clause, romanized ones included, holds only the words around it to the locale's language. A turn written in another language still fails. Signed-off-by: Yev Meyer <ymeyer@nvidia.com>
A probe's user_context line opens the first user message the assistant sees, as the assistant's app would attach it, and the stored transcript keeps it. The simulated user's follow-up prompt and the follow-up gate read their history from that transcript, so the user saw its own opening as "[2026-11-06 · Austin, Texas, USA]" followed by its text, a line it never typed. Both now read the first user message as the user wrote it. The stored transcript, the assistant and the assistant judge keep the line. Signed-off-by: Yev Meyer <ymeyer@nvidia.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Related issue
None.
What changed and why
Two follow-ups to how the simulated user's messages are instructed and shown.
general_open_ended'slanguage_translationtopic andgeneral_educational's language-learning subject drew retries, language violations and gate exhaustion. Every language rule now leaves such text alone: the user's instruction and per-turn requirement say that text it asks about, such as a phrase to translate or a sentence to have corrected, may stay in its own language, and every gate clause, romanized ones included, holds only the words around it to the locale's language. A turn written in another language still fails.user_contextline, such astool_calling's[2026-11-06 · Austin, Texas, USA], opens the first message the assistant sees, and the stored transcript keeps it. The follow-up prompt and the follow-up gate read their history from that transcript, so the user saw a line it never typed at the top of its own opening. Both now read the first user message as the user wrote it; the transcript, the assistant and the assistant judge keep the line.docs/engine/AUTHORING_A_PROBE.mddescribes the exception in its account of language guidance.How it was verified
New tests fail on the code before the change and pass after it: every locale's instruction, per-turn requirement and gate clause allow the text a turn asks about (
test_user_language.py, 33 locales), and with a context line set, neither the user's follow-up prompts nor the user-turn gates contain it (test_simulation.py).On each commit's own tree:
make check-all, the pre-commit hooks on its files andpytest --collect-only(5233, then 5234 tests collected).At the head, on its exact tree in a fresh environment:
Live, against a custom provider config, on these two commits over the previous
main(the identity probe changes that landed since touch none of their files): the end-to-end health check with its default probe and locale, and with thetool_callingprobe in English, the evaluator answering every request in both:Results impact
Every probe's simulated user hears the new allowance in its instructions, and every locale's user-turn gate carries it, so generated user turns can change in any probe; conversations on translation or language practice should see fewer language retries and failed openings. In conversations from a probe with a
user_contextline (tool_calling), the simulated user's follow-ups are written without that line in view. No probe's prompt version changes.